Resolved -
On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.
On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.
In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.
We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.
We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.
May 26, 10:17 UTC
Update -
We continue to see signs of recovery.
May 26, 10:09 UTC
Update -
Issues is operating normally.
May 26, 09:51 UTC
Update -
Git Operations is operating normally.
May 26, 09:46 UTC
Update -
API Requests is operating normally.
May 26, 09:44 UTC
Update -
Copilot is operating normally.
May 26, 09:43 UTC
Update -
Packages is operating normally.
May 26, 09:43 UTC
Update -
Actions is operating normally.
May 26, 09:42 UTC
Update -
Packages is experiencing degraded performance. We are continuing to investigate.
May 26, 08:39 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
May 26, 08:26 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
May 26, 08:25 UTC
Update -
We are continuing to investigate degraded performance.
May 26, 07:53 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
May 26, 07:35 UTC
Investigating -
We are investigating reports of degraded performance for API Requests and Git Operations
May 26, 07:21 UTC